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1 Introduction 



The recovery of torpedoes on the various NUWES test ranges is an im- 
portant problem to the U.S. Navy. The recovery operation is carried on 
undersea with the aid of digging equipment and underwater video cam- 
era equipped with artificial lights. The digging operation is controlled and 
monitored on video monitors at the surface on board the recovery vessel. 

The recovery of torpedoes is not a simple problem. Torpedoes are some- 
times buried in the sediment and are difficult to locate. In addition, the 
recovery equipment in its attempt to dig out the torpedo stirs up sediment 
which visually obscures the object of interest and impedes the recovery 
operation. Since the human operator carrying on the recovery operation 
reUes heavily on the video image received from the imderwater camera and 
displayed on the monitor, improving the quality and intelligibihty of the 
video image in this tturbid water viewing environment has the potential to 
significantly increase the efficiency of the difficult recovery operation. 

An effort to process images in turbid water viewing environment for 
possible improvement in image quality and intelligibility is in progress at 
the Naval Postgraduate School under the direction of Professor Charles 
Therrien. As a part of this effort, over the past six weeks we investigated 
the performance and computational requirements of an existing algorithm. 
We also developed new methods, investigated their computational require- 
ments, and studied their expected performance. Finally, we performed 
some very preliminary study on issues related to the possible real time im- 
plementation of these algorithms. In this report, we describe the results of 
these efforts. 
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2 Adaptive Filtering 

2.1 Basic Algorithm 

Observation of video images in a turbid water environment shows that the 
contrast of images is often reduced significantly for a variety of reasons 
including the presence of sediment that is stirred in a digging operation. 
One approach that was previously considered at the Naval Postgraduate 
School and which we explored further as a part of this effort is to apply a 
contrast enhancement algorithm. The specific algorithm used is shown in 
Figure 1 . 

In the figure, /(ni,ri2) is a frame (N x N pixels) of the video data, 
/£,(ni, TI2) is the local luminance mean which is obtained by low-pass filter- 
ing /(ui,n2), and ///(ni,n2) is the local contrast obtained by subtracting 
/c,(ni,n2) from /(ui, n2). The low-pass filtering operation is performed 
by local averaging over an M x M pixel region. The processed local con- 
trast /f/(ni,n2) is obtained by multiplying by the contrast en- 

hancement factor k{fL). The processed local luminance mean ^(^1,^2) is 
obtained by applying a point non-linearity function to ^2). The pro- 

cessed image p{rii,n2) is obtained by combining /jy(ni,n2) and /l(^15»^2)- 

The algorithm in Figure 1 is capable of modifying the local contrast 
as a function of the local luminance mean and modifying the local lumi- 
nance mean. The algorithm has been [ 1 ] successfully applied previously to 
improving the contrast of optical images talcen from an airplane through 
varying amounts of cloud cover. Since degradations such as those due to 
the sediment stirred up during the recovery operation appear to reduce the 
image contrast in a maimer similar to the contrast reduction due to cloud 
cover, this particular algorithm was chosen originally for a more detailed 
study. 
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2.2 Parameter Choice and Expected Performance 

The algorithm in Figure 1 has been applied to a limited set of data from 
NUWES. Ftom this study, we have observed that the data generally have a 
certain level of backgroimd noise and this limits the utility of the adaptive 
filtering algorithm. Any contrast enhancement method including the adap- 
tive filtering method we have studied tends to emphasize high frequency 
components and as a result tends to emphasize the backgroimd noise. For 
the video data from NUWES, we have observed that the background noise 
becomes very visible when the contrast enhancement factor ^(/l) is chosen 
above 3. This is in sharp contrast with the cloud cover data to which this 
algorithm was originally applied. The noise level in the cloud cover data 
was sufiB.ciently low so that the contrast enhancement factor in the range 
of 6 8 could be used without any visible noise in the processed image. 

Based on the above considerations, the choice of A:(/l) and the nonlin- 
earity function we recommend is shown in Figures 2 and 3. The function 
^(/l) lies between 1.0 and 3.0, and increases as /j, increases. This feature 
exploits the notion that the same level of noise is more visible in the dark 
regions relative to the bright regions. We also recommend a mask size of 
5x5 pixels for the low-pass filtering operation. 

Since the contrast enhancement factgor k{fi) recommended is less than 
3, a very large amount of contrast enhancement is not possible. However, 
the contrast enhancement by a factor of 2 ~ 3 is still significant and we 
expect that noticeable contrast enhancement is still possible with the rec- 
ommended choice of parameters. 

2.3 Computational Requirements 

All our discussions on computational requirements in this and future sec- 
tions are based on the assumption that /(ni , 723 ) is an image frame of TV x iV 
pixels, the low-pass filtering operation is a simple average over a region of 
M X M pixels, and R frames axe processed per second. 

The major computations involved in processing one image frame using 
the adaptive filtering algorithm are as follows: 



5 







L 



r 



KtcohvwiervdtJ clwice 'j 




-Pl 




3. ^ccy^y/ii/icLjcI ClioiCi o'f //on/z/i^r/'ty 



7 





Additions 


Multiplications 


Table Look-ups 


Low-pass filtering 








Subtraction of /l(^i,^ 2 ) 








H/l) 






N 2 


Multiplication with ^(/l) 




^2 




Nonlinearity 








Addition of /ff (ni, 712) 








and fL(ni,n2) 








Total 









From the above result, 



Computations/sec: (M^ + 2)N'^R additions, 2N‘^R multiplications, 

2N‘^R table look-up operations 



As an example, when N = 512, M = 5, and R = 30 (full video rate), we 
require approximately 200M additions, 15M multiplications, and 15M table 
look-up operations per second. As another example, when N = 512, M = 5, 
and = 1 (one frame/sec), we require approximately 7M additions, 
multiplications and |M table look-up operations per second. 

It may be possible to reduce computations that arise from the low-pass 
filtering operation,- which is the major computational requirement in the 
algorithm. The low-pass filtered version /i,(ni,n 2 ) has a low-pass charac- 
teristic and we may be able to undersample it. If we imdersample ^^ 2 ) 

by a factor of S X S, we can reduce the computations required in the low- 
pass filtering operation by a factor of at the expense of an increase of 
approximately additions and iV^ multiplications per frame required for 
interpolation of /L(ni,n 2 ). There will be some decrease in performance, 
but the level of performance decrease is not expected to be too serious. 
We recommend that future study include a study on this trade-off between 
performance decrease and reduction in computations. 

To determine the storage requirements, we assume that the video data 
are raster- scanned as in the NTSC signal. We also assume that one memory 
unit contains one pixel intensity, which is typically represented with 8 bits 
(1 byte). The storage requirements are as follows: 
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/(ni,n2) 


MN memory units 


/L(ni,n2) 


MN memory units 


Kh) 


256 memory units 


Nonlinearity 


256 memory units 



Total : 2MN + 512 memory units 

Prom the above result, 

Storage Requirements : 2MN + 512 memory units 

For N = 512 and M = 5, the algorithm requires about 6K memory 
units. This memory requirement is not an issue in the real time implemen- 
tation of the adaptive filtering algorithm. 
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3 Modified Adaptive Filtering 

3.1 Basic Algorithm 

To accoxint for the noise visibility problem associated with the adaptive 
filtering method, one approach proposed and studied by Franco [2] is to 
enhance the contrast using the adaptive filtering algorithm and then to 
apply a noise reduction system to reduce the backgroimd noise that has 
been accentuated. Even though this approach is interesting, it has several 
problems. In a typical noise reduction system, the local luminance mean 
/i,(ni,n 2 ) is often computed and cascading the adaptive filtering method 
with a noise reduction system requires computation of ^(^ 1 ,^ 2 ) twice. In 
addition, a noise reduction system often requires an estimate of the noise 
variance. The noise variance depends on the contrast enhancement factor 
^(/l) used in the adaptive filtering algorithm, and this complicates the 
design of the noise reduction system. 

A simple method which in a sense integrates contrast enhancement with 
noise reduction has been developed. We’ll refer to this method as modified 
adaptive filtering algorithm. The algorithm is shown in Figure 4. The 
main difference between this algorithm and the adaptive filtering method 
discussed in Section 2 is that the contrast enhancement factor k is now a 
function of the local Iximinance mean fi and the local variance crj. The 
local variance can be computed approximately from ///(ni,n 2 ) by 

. + n2 + -^^ 

cr5(ni,n2) = — fliKh) 

k2=ri2-^^ 

where L is assumed to be an odd integer and the region over which cr'j is 
computed has size o£ L x L pixels. The normalization factor can be 
accounted for in determining k(fL,crj) and therefore can be ignored. 

When a} is small, the area is likely to correspond to a region with little 
signal component (uniform background region) and contrast enhancement 
in the region is likely to boost only the background noise. When crj is very 
large, the signal component may have large energy and a very large contrast 
enhancement may not be necesseiry. When cr^ is in the mid-range, a weak 
signal component is likely to be present and a large contrast enhancement 
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is likely to be beneficial. Even though the noise in this region will be 
accentuated, the region involved is only a fraction of the image frame and 
the presence of the accentuated signal component can ma^k the noise to 
some extent. The dependence of k(fL,cr^) on crj allows us to modify the 
contrast as a function of crj. 

3.2 Parameter Choice and Expected Performance 

The algorithm in Figure 4 is currently being implemented by Lieutenant 
Roberto Ventura. At the time of this report, we do not have results of 
applying this method to the data from NUWES. It is expected, however, 
to solve the noise visibility problem associated with the adaptive filtering 
method to some extent. 

Choosing the parameters of the algorithm requires careful evaluation 
of the processed images. As an initial starting point, we recommend the 
parameter choice shown in Figures 5 and 6. The choice of a separable 
function k{fL,aj) = k\{fL)k 2 {crj) is due to the desire to decouple the effects 
of fi, and <rj on k{fi,,<Tj) and reduction in the required storage. If storage 
is not a problem, fc(/t,, crj) can be precomputed and stored. The choice of 
kiifi) is based on the same considerations as in Section 2.2. The choice 
of k 2 {(jj) is chosen based on several considerations. When crj is small, 
there is hkely to be httle signal component and large contrast enhancement 
will only result in noise enhancement. When aj is very large, the signal 
component is already very strong and there is not much need for a large 
contrast increase. 



3.3 Computational Requirements 

The modified adaptive filtering method in Figiure 4 is more expensive com- 
putationally than the adaptive filtering method in Figure 1. The additional 
computations per second are 

Computation of Variance; L^N^R multiply/ adds. 

Computation of ^(/t, crj): N^R multiplications, table lookups. 

Combining this result with the results in Section 2.3, 

Computations /sec: (M^ -f- 2)N'^R additions, 3 N^R multiplications. 
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L^N^R multiply/adds, ZN^R table look-up operations 



K we assTime that L = M and one multiply/add operation, one add 
operation, and one multiply operation take the same computation time, the 
computational requirements of the modified adaptive filtering algorithm is 
approximately twice that of the adaptive filtering algorithm. 

The storage requirements axe essentially the same as that of the adaptive 
method. The small increase is due to the storage of k 2 {<rj). The storage 
requirements axe 

Storage Requirements: 2MN + 768 memory units 
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4 Temporal Filtering 

4.1 Basic Algorithm 

Careful observation of the video data from NUWES shows that some form 
of frame averaging may result in image enhancement. Specifically, in a typ- 
ical torpedo recovery operation, the underwater camera is often stationary 
or moves slowly. As a result, objects of interest such as a torpedo displayed 
on the monitor do not appear to change much from one frame to the next 
frame. However, the elements such as water bubbles and stirred- up sedi- 
ment that degrade the visibility of objects of interest appear to change quite 
rapidly. Some form of frame averaging, therefore, has potential to add the 
signal component constructively while adding signals from the degrading 
sources destructively. This can result in improvement of the signal to noise 
ratio (SNR). 

Frame averaging can involve storage of the frames involved, which can 
increase the storage requirements drastically. One method that performs 
frame averaging without requiring storage of more than one frame at a time 
is first order recursive temporal filtering. Consider one particular pixel. Let 
/(n) denote the intensity of the image at the nth frame at that particular 
pixel. Note that the variable “n” is a time variable, not a spatial variable. 
The processed image p(n) in the first order rectirsive temporal filtering is 
given by 

p{n) = (1 - a) • p(n - 1) -1- a • /(n), 

where “a” is a constant between 0 and 1. As /(n) is received and once 
p(n) is computed, p(n — 1) is no longer needed and can be replaced with 
p(n). At any given time, therefore, we need to store only one frame of data. 
The system fimction H{z) and unit sample response of the system h{n) are 
given by 

a 

^ 1 - (1 - a)z-^ 

and 

h{n) = a(l — a)”u(n). 

As “a” approaches 0, many frames are averaged. When a = 1, no temporal 
filtering is performed. 
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Many variations of the above recursive filtering method axe possible. 
For example, by initializing p{n) once in a while, choosing “a” close to 
zero, and normalizing the result, a result essentially the same as simple 
FIR filtering can be obtained. We also note that the contrast enhancement 
methods discussed in Sections 2 and 3 can also be applied to the temporally 
filtered images, if desired. 

4.2 Parameter Choice and Expected Performance 

A sequence of image frames that have been digitized are not available and 
we were not able to process the data using the temporal filtering algorithm 
discussed above. To the extent that our assumption that objects of inter- 
est do not change rapidly between consecutive frames while the degrading 
sources do is valid, the algorithm is expected to perform very well. 

To see the expected performance improvement in an ideal environment, 
suppose /(n) can be expressed as 

f(n) = s(n) -H w(n) 

where s{n) is the signal whidi is constant independent of n and w{n) is 
zero- mean white noise with variance of cr^. The processed image p{n) can 
be expressed as 

p(n) = s{n) + Wp{n) 

where Wp{n) is zero-mean white (in the spatial domain) noise with vari- 
ance of • <7^. When a = 0.2, the noise variance reduction is by 

approximately a factor of 10, corresponding to lOdB SNR improvement. 

In practice, of course, the assumptions made in the above analysis will 
not be valid. The signal, for example, will change as a function of time 
and this will cause signal blurring. There is potential for significant image 
enhancement, however, and we recommend that future studies include the 
application of temporal filtering to the data from NUWES. Our initial 
recommendation for the choice of “a” is 0.2. 

4.3 Computational Requirements. 

The computations involved are iV^ additions and 2iV^ multiplications per 
frame. Therefore, 



17 



Computations/sec: N^R additions, 2N^R multiplications 



When N = 512 and R = 30, the computational requirements are 7^M 
additions and 15M multiplications per second. 

The storage requirement is the storage of one image frame and therefore, 

Storage requirement: memory units 
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5 Pseudo Color Representation 

It is well known that the human visual system is quite sensitive to color. 
The number of distinguishable intensities, for example, is much less than 
the number of distinguishable colors and intensities. As a result, when a 
black and white image is displayed using color, the result may not be a 
natvural looking image but the contrast of the image may be considerably 
improved. In addition, color images are generally much more pleasant to 
look at than black and white images. 

Mapping a black and white image /(ni,ri2) to a color image involves 
generation of the red component fR{ni,ri2), the green component /g(^ 1) ^2) 
and the blue component /b(^i,^ 2)- Designing the specific treinsformation 
that maps f(ni,n2) to fR(ni,n2), /g(»^1j^ 2) aJid /b(^ 1)*^2) is limited only 
by one’s artistic imagination, and involves a fair amoimt of trial and error. 
One recommendation we have for the design of the transformation table 
is that similar colors be used for similar pixel intensities. Otherwise, the 
resulting color image will appear noisy. Another recommendation is that 
low intensities be mapped to blue and high intensities be mapped to red. 
The intensities in between can perhaps be mapped following the rainbow 
colors. It is well known that people perceive blue as “cold” or “dark” and 
perceive red as “hot” or “bright”. 

Once the transformation table is designed, processing involves only N'^R 
table look-up operations per second and storage requirement for the table 
is minimal. Allowing table look-up operations prior to image display is a 
common feature in commercially available real time video data digitization 
and display systems. Even though the level of performance improvement 
is not clear, it is a worthwhile approach to be explored in future study. 
Pseudo-color operation, of course can be performed in addition to other 
methods discussed in previous sections. 
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6 Heirdware Implementation 

To apply the algorithms discussed in previous sections to enhance- the ef- 
ficiency of the torpedo recovery operation, their real time implementation 
is essential. In this section we discuss some very preliminary studies we 
performed to determine the feasibility of implementing the algorithms in 
real time at a reasonable cost. 

One cost-effective approach to real time implementation is to interface 
a signal processing (SP) chip to a personal computer (PC) which has been 
interfaced to a unit such as the PC Vision board that is capable of digitizing 
and displaying the video data in real time, as shown in Figure 7. Using a PC 
as an overall control unit and interfacing an image digitization and display 
unit such as the PC Vision board is a very cost-effective way to acqxiire 
and display video data. The algorithms we wish to implement in real time 
require a very large number of arithmetic operations and the SP chip is 
necessary to perform the arithmetic operations very fast. The adaptive 
filtering method discussed in Section 2 has already been implemented [3] 
using the approach in Figure 7. With a NEC /iPD77230 chip and PC-9800, 
processing one image frame using the adaptive filtering method discussed 
in Section 2 with N=256 and M=17 requires approximately 3 seconds. 

In studying the feasibifity of real time implementation, a number of is- 
sues have to be considered. In this very preliminary study, however, we 
have made a variety of simplifying assumptions. Specifically, we assume 
that storage requirement is not an issue in real time implementation. Since 
the storage requirement for data is in the order of memory units or 
less, this assumption is reasonable. We have also assumed that the compu- 
tational time required for each of an add, a multiply, and a multiply/add 
operation is the same. As floating point arithmetic operations are becoming 
more common, this assumption also appears reasonable. We’ll refer to an 
addition, a multipUcation, or a multiply/add operation as one arithmetic 
operation. We denote the computation time for one arithmetic operation 
by c sec. For currently available SP chips such as Texas Instrument’s most 
recent TMS320 family chip, the computation time “c” is in the order of 100 
nanosec (10~® sec). We also assume that the computation time required 
for all arithmetic operations is a fraction (p%) of the total amount of time 
required for a given algorithm. The fraction “p” % depends on a number 
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of factors such as data transfer time, algorithm complexity, the specific SP 
chip, PC, and interface used. We’ll consider two different reasonable values 
of p in our analysis. 

Under the assumptions and notations discussed above, we have com- 
puted the amount of time required for processing 1 second of video data 
for various choices of the parameters. The results are shown in Table 1. If 
the required processing time is less than 1 second, it implies that real time 
implementation is possible. For the specific choices of the parameters used 
in the table, none of the cases can be implemented in real time. However, 
some of the cases such as temporal filtering are not far from real time. 
There are a variety of ways to reduce computational requirements. For ex- 
ample, as we discussed in Section 2, we may be able to reduce computations 
reqmred for the low-pass filtering operation by under-samphng /L(rii,n 2 ). 
It is also possible to increase the computational speed. For example, the 
computation time required for an arithmetic operation is becoming smaller 
as new SP chips are introduced. In addition, the algorithms have very sim- 
ple structures and we can process the data in parallel using more than one 
SP chip. 

In summary, ovu: preliminary study indicates that real time implemen- 
tation of the algorithms discussed in previous sections at a reasonable cost 
is not a simple task due to a very large amount of data involved. However, 
it appears to be within a reachable goal with the hardware technology that 
is currently available or will shortly be available. 



TABLE I 



Algorithms 




Choice of c and p 






c — 100 nsec 
p = 25% 


c = 100ns 
p = 50% 


c = 200 nsec 
p = 25% 


c = 100 nsec 
p = 50% 


Adaptive Filtering 
iV = 512,M = 5,iZ = 30 


90 sec 


45 sec 


180 sec 


90 sec 


Adaptive Filtering 
iV = 512,M = 5,i2 = l 


3 sec 


1.5 sec 


6 sec 


3 sec 


Modified Adaptive Filtering 
N = bl2,M = 5,L = b,R = Z0 


180 sec 


90 sec 


360 sec 


180 sec 


Modified Adaptive Filtering 
N = bl2,M = b,L = 5,R = l 


6 sec 


3 sec 


12 sec 


6 sec 


Temporal Filtering 
N = bl2,R = 3Q 


9 sec 


4.5 sec 


18 sec 


9 sec 
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